Extracting low-dimensional features by means of Deep Network Architectures

Inspired by nature, arti cial neural networks are an attempt to create a learning machine that is comparable to real brains. Due to the complexity of brains, and a lack of understand there of, arti cial neural networks are only remotely related to their natural counterparts. Also, in practice it has proven intractable to work with arbitrarily large neural networks as they are hard to control and even harder to learn, since it is computationally expensive and the result may still be far from optimal. Because of these reasons neural networks up to now have been rather limited in size and capability. With a newly proposed algorithm called pretraining it is now possible to train neural networks with a very complex structure, i.e. so called deep network. This algorithm is applied before an error correction algorithm, e.g. backpropagation. It carefully tunes the neural connection weights of a network such that they reach a good starting point for error backpropagation and converge quickly towards a good solution. In this thesis we will concentrate on a special kind of neural network for unsupervised learning, so-called auto-associative neural networks or simply autoencoder. These networks learn an identity mapping for the data in the input space. While the network learn this mapping it forces the data through a bottle-neck layer. Thus, the network has to learn a code that categorizes the data according to its intrinsic structure without the use of any label information. We will suggest a framework for successful feature extraction using deep autoencoders and will use these extracted features to perform classi cation of various types of real-life data. Also, a new approach to model selection for auto-associative networks will be proposed. The advantage of this method compared to conventional methods is the huge economization of training time, which can be immense if the network is very deep and conists of millions of neural connections.